To address the problems like static nature, unfairness and complexity in Selfish Misbehavior (SM) processing mechanism of Medium Access Control (MAC) protocol of Mobile Ad Hoc NETwork (MANET), an optimization algorithm for SM was proposed. By using optimization theory and feedback theory, the Optimal Access Probability (OAP) was conducted through the utilization of historical samples, realizing the dynamic change of parameters to improve static nature. Then, all nodes in the network were set to use the OAP at the given period, thus the fairness index of the network was promoted. Finally, linear iteration mechanism was adopted to avoid the increase of complexity. On basis of the above, stability and effectiveness of the proposed algorithm were proved theoretically by Lyapunov algorithm and global stable point. Experimental results show that, by the proposed algorithm, the number of SM decreases by 30%-50%, the end-to-end delay brings down 8-10 ms, the throughput increases about 0.5 Mb/s, the fairness index raises by 0.05, while the control overhead remains unchanged, all of which indicates that the performance of the SM processing mechanism has been improved.